ci: adopt benchmark-qualified Koru judge - #4
Open
tom-sapletta-com wants to merge 7 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
google/gemini-3.1-pro-previewverifyjobLocal validation
npm run verify:workflows— PASSnpm run verify— PASS (335 tests, 334 passed, 1 JDK-dependent skip)make governance— expected inherited ticket-019 findings only; no ticket-018 scope/secret findingTimeout and failure policy
Live evidence
Repository-scoped
OPENROUTER_API_KEYwas rotated throughghstdin after explicit user authorization. Dispatch30714664770reachedopenrouter/google/gemini-3.1-pro-previewfor both PR #3 TypeScript files without authentication or provider-limit errors. Both file-level verdicts werepass; that run exposed the parser and aggregate-policy defects repaired by this PR.Repaired live evidence
Dispatch
30746421293reviewed PR #3 range2e87205..6b79527in 1m24s. The attested report records Gemini, both expected TypeScript files, verdictpass, exit 0, 2/2 passed, zero blocking/parser/provider findings, and five retained advisory semantic observations. Vallm original exit 2 remains recorded before policy normalization.Remaining blockers
Follow-up failure-path validation
LLM boundary verified transitively from 9 deterministic entrypoints across 38 modules.
Module boundaries verified: 114 modules, 521 internal imports, no cycles, core is independent.
Environment contract verified: 75 code/Docker variables, 75 documented keys, no duplicates.
Workflow YAML verified: 2 file(s), no duplicate top-level keys.
{"filesChecked":19,"untrackedInputsChecked":0,"status":"ok"}
{"structuredCalls":7,"rawCalls":0,"status":"ok"}
{"schema":"schemas/document-extraction-response.schema.json","status":"ok"}
TAP version 13
[t2c:a2a] listening on 127.0.0.1:38195
[t2c:a2a] listening on 127.0.0.1:37603
Subtest: A2A governed-intake skill completes accepted commands and rejects domain failures
ok 1 - A2A governed-intake skill completes accepted commands and rejects domain failures
duration_ms: 133.049905
...
[t2c:a2a] listening on 127.0.0.1:35497
Subtest: A2A v1.0 card, versioning, task methods and cursor pagination are coherent
ok 2 - A2A v1.0 card, versioning, task methods and cursor pagination are coherent
duration_ms: 140.624917
...
[t2c:a2a] listening on 127.0.0.1:41177
Subtest: A2A bearer authentication is declared with v1 security objects and enforced
ok 3 - A2A bearer authentication is declared with v1 security objects and enforced
duration_ms: 32.579244
...
[t2c:a2a] listening on 127.0.0.1:41259
[t2c:a2a] listening on 127.0.0.1:32825
[t2c:a2a] listening on 127.0.0.1:43233
Subtest: A2A file task store survives restart and preserves idempotency across replicas
ok 4 - A2A file task store survives restart and preserves idempotency across replicas
duration_ms: 110.351699
...
Subtest: the A2A bind default is loopback and a wildcard bind is not mistaken for it
ok 5 - the A2A bind default is loopback and a wildcard bind is not mistaken for it
duration_ms: 1.834447
...
Subtest: Go adapter records package, imports, types, functions and methods
ok 6 - Go adapter records package, imports, types, functions and methods
duration_ms: 191.156694
...
Subtest: Go facts are deterministic observations, not inferences
ok 7 - Go facts are deterministic observations, not inferences
duration_ms: 144.172903
...
Subtest: Go adapter marks exported symbols and reports calls in scope
ok 8 - Go adapter marks exported symbols and reports calls in scope
duration_ms: 113.207635
...
Subtest: Go extraction is skipped without cost when a tree holds no Go sources
ok 9 - Go extraction is skipped without cost when a tree holds no Go sources
duration_ms: 14.56097
...
Subtest: A missing Go toolchain degrades to a warning instead of failing the run
ok 10 - A missing Go toolchain degrades to a warning instead of failing the run
duration_ms: 11.266279
...
Subtest: Rust adapter records uses, types, functions, methods, values and calls
ok 11 - Rust adapter records uses, types, functions, methods, values and calls
duration_ms: 104.750893
...
Subtest: Java adapter records packages, imports, types, fields, methods and calls
ok 12 - Java adapter records packages, imports, types, fields, methods and calls # SKIP JDK not installed
duration_ms: 8.966518
...
Subtest: Java and Rust adapters skip toolchain startup when no matching sources exist
ok 13 - Java and Rust adapters skip toolchain startup when no matching sources exist
duration_ms: 21.042575
...
Subtest: Missing Java and Rust toolchains degrade to explicit warnings
ok 14 - Missing Java and Rust toolchains degrade to explicit warnings
duration_ms: 51.660978
...
Subtest: PHP syntax adapter records namespaces, imports, types, functions, methods and calls
ok 15 - PHP syntax adapter records namespaces, imports, types, functions, methods and calls
duration_ms: 101.635199
...
Subtest: PHP adapter skips runtime startup when no PHP source exists
ok 16 - PHP adapter skips runtime startup when no PHP source exists
duration_ms: 16.947268
...
Subtest: Missing PHP runtime degrades to an explicit warning
ok 17 - Missing PHP runtime degrades to an explicit warning
duration_ms: 9.744386
...
Subtest: Invalid PHP syntax is reported without aborting extraction
ok 18 - Invalid PHP syntax is reported without aborting extraction
duration_ms: 75.710835
...
Subtest: AST extractor reads TypeScript and Python facts
ok 19 - AST extractor reads TypeScript and Python facts
duration_ms: 110.625447
...
Subtest: CLI command help is successful and non-mutating
ok 20 - CLI command help is successful and non-mutating
duration_ms: 2375.182123
...
Subtest: TypeScript and Python CLIs execute the same intake command/query handler
ok 21 - TypeScript and Python CLIs execute the same intake command/query handler
duration_ms: 1629.798729
...
Subtest: CLI summarize exposes deterministic, prefer-llm and require-llm modes
ok 22 - CLI summarize exposes deterministic, prefer-llm and require-llm modes
duration_ms: 4100.87357
...
Subtest: CLI propose-todo, render-todo and apply-todo return JSON and preserve a no-op TODO
ok 23 - CLI propose-todo, render-todo and apply-todo return JSON and preserve a no-op TODO
duration_ms: 3147.502079
...
Subtest: CLI watch reads TASK.md by default, disables summary LLM and reacts to a live file change
ok 24 - CLI watch reads TASK.md by default, disables summary LLM and reacts to a live file change
duration_ms: 2086.17955
...
Subtest: proposeCodeChangePlans materialises grounded plans from PLANNED_NOT_IMPLEMENTED
ok 25 - proposeCodeChangePlans materialises grounded plans from PLANNED_NOT_IMPLEMENTED
duration_ms: 31.288063
...
Subtest: code-change title preserves the leading action of a compound intent
ok 26 - code-change title preserves the leading action of a compound intent
duration_ms: 2.31415
...
Subtest: proposeCodeChangePlans is deterministic for the same evidence
ok 27 - proposeCodeChangePlans is deterministic for the same evidence
duration_ms: 3.980339
...
Subtest: a plan creates a missing file and modifies an existing one
ok 28 - a plan creates a missing file and modifies an existing one
duration_ms: 11.564583
...
Subtest: a plan never invents a repository-root file from a bare filename
ok 29 - a plan never invents a repository-root file from a bare filename
duration_ms: 3.953313
...
Subtest: a plan whose every path is bare and missing is withheld entirely
ok 30 - a plan whose every path is bare and missing is withheld entirely
duration_ms: 1.724869
...
Subtest: a plan never targets a location outside the repository tree
ok 31 - a plan never targets a location outside the repository tree
duration_ms: 0.312334
...
Subtest: the repository probe never proposes creating a file outside the root
ok 32 - the repository probe never proposes creating a file outside the root
duration_ms: 1.263122
...
Subtest: bounded plan sets prefer explicit TODO work over historical changelog audit
ok 33 - bounded plan sets prefer explicit TODO work over historical changelog audit
duration_ms: 3.550929
...
Subtest: evaluateCodeChangeAcceptance passes when targeted diagnostics clear
ok 34 - evaluateCodeChangeAcceptance passes when targeted diagnostics clear
duration_ms: 5.921532
...
Subtest: evaluateCodeChangeAcceptance fails while the plan is still open
ok 35 - evaluateCodeChangeAcceptance fails while the plan is still open
duration_ms: 1.819317
...
Subtest: Plans without repository paths are not invented
ok 36 - Plans without repository paths are not invented
duration_ms: 0.986083
...
Subtest: Non-repository paths are ignored instead of aborting code-change planning
ok 37 - Non-repository paths are ignored instead of aborting code-change planning
duration_ms: 1.280131
...
Subtest: Acceptance rejects ungrounded paths, missing provenance and inconsistent verdicts
ok 38 - Acceptance rejects ungrounded paths, missing provenance and inconsistent verdicts
duration_ms: 4.742995
...
Subtest: createCodeChangeSourcePatch is deterministic and path-bound
ok 39 - createCodeChangeSourcePatch is deterministic and path-bound
duration_ms: 4.745655
...
Subtest: applyUnifiedDiffToText creates and modifies files from hunks
ok 40 - applyUnifiedDiffToText creates and modifies files from hunks
duration_ms: 0.728403
...
Subtest: applyCodeChangeSourcePatch requires approval and is idempotent
ok 41 - applyCodeChangeSourcePatch requires approval and is idempotent
duration_ms: 14.276191
...
Subtest: applyCodeChangeSourcePatch preflights diffs and refuses symlink escapes
ok 42 - applyCodeChangeSourcePatch preflights diffs and refuses symlink escapes
duration_ms: 5.517787
...
Subtest: createCodeChangeSourcePatchSet covers every plan
ok 43 - createCodeChangeSourcePatchSet covers every plan
duration_ms: 2.901765
...
Subtest: createCodeChangeReviewPatch is hash-stable and lists grounded paths
ok 44 - createCodeChangeReviewPatch is hash-stable and lists grounded paths
duration_ms: 2.789541
...
Subtest: CLI proposes and evaluates a grounded code-change plan through persisted JSON
ok 45 - CLI proposes and evaluates a grounded code-change plan through persisted JSON
duration_ms: 3226.923974
...
Subtest: isUsefulCodeChangePath rejects vendored, binary and analysis dump paths
ok 46 - isUsefulCodeChangePath rejects vendored, binary and analysis dump paths
duration_ms: 0.475118
...
Subtest: proposeCodeChangePlans skips diagnostics that only name junk paths
ok 47 - proposeCodeChangePlans skips diagnostics that only name junk paths
duration_ms: 0.925965
...
Subtest: Published code-change JSON schemas require provenance, risk and rollback
ok 48 - Published code-change JSON schemas require provenance, risk and rollback
duration_ms: 1.832967
...
Subtest: participant registry maps stable IDs to Git/A2A identifiers without display-name guessing
ok 49 - participant registry maps stable IDs to Git/A2A identifiers without display-name guessing
duration_ms: 52.365262
...
Subtest: participant registry rejects ambiguous external identifiers
ok 50 - participant registry rejects ambiguous external identifiers
duration_ms: 0.672329
...
Subtest: trusted intake persists roles across tickets and fails closed without rejected writes
ok 51 - trusted intake persists roles across tickets and fails closed without rejected writes
duration_ms: 117.61495
...
Subtest: agents cannot create human projections and a damaged event hash fails verification
ok 52 - agents cannot create human projections and a damaged event hash fails verification
duration_ms: 25.243968
...
Subtest: TypeScript and dependency-free Python codecs share golden bytes and preserve unknown fields
ok 53 - TypeScript and dependency-free Python codecs share golden bytes and preserve unknown fields
duration_ms: 80.695325
...
Subtest: communication enrichment preserves runtime identity, source, ticket and epistemic class
ok 54 - communication enrichment preserves runtime identity, source, ticket and epistemic class
duration_ms: 66.589084
...
Subtest: communication enrichment corrects one rejected structured response without weakening validation
ok 55 - communication enrichment corrects one rejected structured response without weakening validation
duration_ms: 15.570318
...
Subtest: communication prefer-llm fallback is explicit and require-llm rejects
ok 56 - communication prefer-llm fallback is explicit and require-llm rejects
duration_ms: 20.178406
...
Subtest: project/ communication is attributed per human and agent and checked against Git evidence
ok 57 - project/ communication is attributed per human and agent and checked against Git evidence
duration_ms: 178.309318
...
Subtest: governance user-* and ai-* files become typed participant intent without ingesting ticket evidence
ok 58 - governance user-* and ai-* files become typed participant intent without ingesting ticket evidence
duration_ms: 27.681578
...
Subtest: unstructured governance participant content is rejected with an owner-specific migration warning
ok 59 - unstructured governance participant content is rejected with an owner-specific migration warning
duration_ms: 4.560874
...
Subtest: opposite wording about different explicit files is not treated as an intent conflict
ok 60 - opposite wording about different explicit files is not treated as an intent conflict
duration_ms: 7.967587
...
Subtest: missing response owners use explicit role sentinels without inventing participants
ok 61 - missing response owners use explicit role sentinels without inventing participants
duration_ms: 11.718646
...
Subtest: communication extractor reports unresolved identity instead of inventing an actor
ok 62 - communication extractor reports unresolved identity instead of inventing an actor
duration_ms: 5.699314
...
Subtest: communication extractor ignores generic generated analysis under project/
ok 63 - communication extractor ignores generic generated analysis under project/
duration_ms: 11.552685
...
Subtest: configuration converter covers JSON, TOML, Docker and CI workflow declarations
ok 64 - configuration converter covers JSON, TOML, Docker and CI workflow declarations
duration_ms: 40.551884
...
Subtest: configuration converter emits a deterministic file aggregate for an empty configuration
ok 65 - configuration converter emits a deterministic file aggregate for an empty configuration
duration_ms: 5.560812
...
Subtest: splitLines treats a trailing newline as a terminator, not an extra line
ok 66 - splitLines treats a trailing newline as a terminator, not an extra line
duration_ms: 1.689699
...
Subtest: Identical inputs produce no hunks
ok 67 - Identical inputs produce no hunks
duration_ms: 0.596966
...
Subtest: A modified line keeps both sides addressable by original line number
ok 68 - A modified line keeps both sides addressable by original line number
duration_ms: 0.403045
...
Subtest: Pure insertion and pure deletion are not reported as replacements
ok 69 - Pure insertion and pure deletion are not reported as replacements
duration_ms: 0.309081
...
Subtest: Empty-to-content and content-to-empty are handled as block changes
ok 70 - Empty-to-content and content-to-empty are handled as block changes
duration_ms: 0.306701
...
Subtest: Context width controls hunk size
ok 71 - Context width controls hunk size
duration_ms: 0.433603
...
Subtest: Nearby changes merge into a single hunk
ok 72 - Nearby changes merge into a single hunk
duration_ms: 0.329865
...
Subtest: Distant changes stay in separate hunks
ok 73 - Distant changes stay in separate hunks
duration_ms: 0.338893
...
Subtest: Oversized inputs fall back to a bounded block replace
ok 74 - Oversized inputs fall back to a bounded block replace
duration_ms: 0.617573
...
Subtest: Unified output carries a well formed hunk header
ok 75 - Unified output carries a well formed hunk header
duration_ms: 0.497057
...
Subtest: Side-by-side rows pair deletions with insertions
ok 76 - Side-by-side rows pair deletions with insertions
duration_ms: 0.328081
...
Subtest: Unbalanced change runs leave one side empty rather than misaligning
ok 77 - Unbalanced change runs leave one side empty rather than misaligning
duration_ms: 1.111211
...
Subtest: Renderers escape source markup
ok 78 - Renderers escape source markup
duration_ms: 1.490097
...
Subtest: SVG rendering caps rows and reports the remainder
ok 79 - SVG rendering caps rows and reports the remainder
duration_ms: 2.02803
...
Subtest: Reality view keys topics by target and records lane presence
ok 80 - Reality view keys topics by target and records lane presence
duration_ms: 20.325596
...
Subtest: A topic holding declared and observed records is never reported as planned-only
ok 81 - A topic holding declared and observed records is never reported as planned-only
duration_ms: 3.784482
...
Subtest: Reality coverage stays open when a shared path has unrelated capabilities
ok 82 - Reality coverage stays open when a shared path has unrelated capabilities
duration_ms: 2.264223
...
Subtest: Shared-path relations do not collapse unrelated files into one topic
ok 83 - Shared-path relations do not collapse unrelated files into one topic
duration_ms: 3.85859
...
Subtest: Reality view is deterministic for identical input
ok 84 - Reality view is deterministic for identical input
duration_ms: 3.073571
...
Subtest: Reality SVG escapes topic labels
ok 85 - Reality SVG escapes topic labels
duration_ms: 1.794578
...
Subtest: graph diff detects changed source identities, additions and SVG-safe labels
ok 86 - graph diff detects changed source identities, additions and SVG-safe labels
duration_ms: 27.5652
...
Subtest: graph diff is empty for graphs with identical evidence
ok 87 - graph diff is empty for graphs with identical evidence
duration_ms: 2.203028
...
Subtest: file diff emits deterministic unified, SVG and HTML views
ok 88 - file diff emits deterministic unified, SVG and HTML views
duration_ms: 2.315685
...
Subtest: intent-vs-reality builds an explainable SVG and Markdown projection
ok 89 - intent-vs-reality builds an explainable SVG and Markdown projection
duration_ms: 5.745638
...
Subtest: a targetless declaration is filed under the single module it links to
ok 90 - a targetless declaration is filed under the single module it links to
duration_ms: 3.413625
...
Subtest: a declaration touching several modules keeps its own topic
ok 91 - a declaration touching several modules keeps its own topic
duration_ms: 3.621179
...
Subtest: a topic about a document does not absorb the statements written in it
ok 92 - a topic about a document does not absorb the statements written in it
duration_ms: 2.96805
...
Subtest: semantically aligned configuration topics retain their evidence grade
ok 93 - semantically aligned configuration topics retain their evidence grade
duration_ms: 2.5496
...
Subtest: A record claiming line 1 is re-anchored to the line carrying its statement
ok 94 - A record claiming line 1 is re-anchored to the line carrying its statement
duration_ms: 52.353844
...
Subtest: An already correct line is kept and not reported as re-anchored
ok 95 - An already correct line is kept and not reported as re-anchored
duration_ms: 8.094266
...
Subtest: An empty target is backfilled from the statement text
ok 96 - An empty target is backfilled from the statement text
duration_ms: 12.889046
...
Subtest: A target supplied by the model is never overwritten
ok 97 - A target supplied by the model is never overwritten
duration_ms: 9.974896
...
Subtest: An unclassified action and modality are derived from the statement
ok 98 - An unclassified action and modality are derived from the statement
duration_ms: 6.824687
...
Subtest: A classified action from the model wins over the heuristic
ok 99 - A classified action from the model wins over the heuristic
duration_ms: 12.18531
...
Subtest: An action that stays unclassifiable is reported as a missing field
ok 100 - An action that stays unclassifiable is reported as a missing field
duration_ms: 7.441539
...
Subtest: A placeholder object is treated as a gap, not as content
ok 101 - A placeholder object is treated as a gap, not as content
duration_ms: 10.724431
...
Subtest: Every repair is attributable through epistemic.basis
ok 102 - Every repair is attributable through epistemic.basis
duration_ms: 11.498534
...
Subtest: deterministic documentation baseline records headings, code blocks and explicit references
ok 103 - deterministic documentation baseline records headings, code blocks and explicit references
duration_ms: 23.09708
...
Subtest: deterministic documentation preserves Polish prohibition polarity
ok 104 - deterministic documentation preserves Polish prohibition polarity
duration_ms: 9.762369
...
Subtest: deterministic documentation resolves a unique bare filename against the repository
ok 105 - deterministic documentation resolves a unique bare filename against the repository
duration_ms: 4.88998
...
Subtest: documentation prose resolves a bare filename to its repository location
ok 106 - documentation prose resolves a bare filename to its repository location
duration_ms: 4.749655
...
Subtest: a nested checkout does not shadow the repository copy of a documented file
ok 107 - a nested checkout does not shadow the repository copy of a documented file
duration_ms: 8.796265
...
Subtest: AST cache is incremental by path and source content hash
ok 108 - AST cache is incremental by path and source content hash
duration_ms: 36.346025
...
Subtest: AST cache rejects corrupt entries and recomputes authoritative records
ok 109 - AST cache rejects corrupt entries and recomputes authoritative records
duration_ms: 8.407583
...
Subtest: AST cache can be bypassed without changing extraction output
ok 110 - AST cache can be bypassed without changing extraction output
duration_ms: 5.489153
...
Subtest: successful external AST adapter is skipped on a warm manifest hit
ok 111 - successful external AST adapter is skipped on a warm manifest hit
duration_ms: 68.187217
...
Subtest: documentation chunks cache independently while provider calls remain live
ok 112 - documentation chunks cache independently while provider calls remain live
duration_ms: 35.314472
...
Subtest: generated analysis replaces its source root with a stable token
ok 113 - generated analysis replaces its source root with a stable token
duration_ms: 69.09302
...
Subtest: generated analysis root normalization refuses the filesystem root
ok 114 - generated analysis root normalization refuses the filesystem root
duration_ms: 36.309173
...
Subtest: generated analysis rejects references to untracked input
ok 115 - generated analysis rejects references to untracked input
duration_ms: 76.356725
...
Subtest: generated analysis accepts outputs independent of untracked input
ok 116 - generated analysis accepts outputs independent of untracked input
duration_ms: 90.265995
...
Subtest: generated analysis accepts an untracked filename already quoted by tracked evidence
ok 117 - generated analysis accepts an untracked filename already quoted by tracked evidence
duration_ms: 108.966923
...
Subtest: generated analysis rejects temporary paths and unavailable validators
ok 118 - generated analysis rejects temporary paths and unavailable validators
duration_ms: 82.013733
...
Subtest: generated README metadata is synchronized from package.json and stays idempotent
ok 119 - generated README metadata is synchronized from package.json and stays idempotent
duration_ms: 109.305934
...
Subtest: generated README synchronization fails closed when the template drifts
ok 120 - generated README synchronization fails closed when the template drifts
duration_ms: 36.16117
...
Subtest: generated README synchronization rejects output outside the project root
ok 121 - generated README synchronization rejects output outside the project root
duration_ms: 32.778528
...
Subtest: Git extractor emits one record per requested commit
ok 122 - Git extractor emits one record per requested commit
duration_ms: 277.011863
...
Subtest: An empty repository degrades to a warning instead of failing the run
ok 123 - An empty repository degrades to a warning instead of failing the run
duration_ms: 12.066202
...
Subtest: versioned gold dataset reports perfect offline quality and repeated-run stability
ok 124 - versioned gold dataset reports perfect offline quality and repeated-run stability
duration_ms: 177.097726
...
Subtest: gold linking reports exact-target and capability-topic quality separately
ok 125 - gold linking reports exact-target and capability-topic quality separately
duration_ms: 82.710325
...
Subtest: gold capability-topic support is large enough to detect a floor regression
ok 126 - gold capability-topic support is large enough to detect a floor regression
duration_ms: 67.985971
...
Subtest: gold known gaps are measured and kept out of precision and recall
ok 127 - gold known gaps are measured and kept out of precision and recall
duration_ms: 70.063407
...
Subtest: gold reports cross-language positives and hard negatives as a separate cohort
ok 128 - gold reports cross-language positives and hard negatives as a separate cohort
duration_ms: 73.882323
...
Subtest: gold diagnostics separate a false DONE claim from an evidenced one
ok 129 - gold diagnostics separate a false DONE claim from an evidenced one
duration_ms: 61.394025
...
Subtest: gold v1 stays evaluable after the v2 contract extension
ok 130 - gold v1 stays evaluable after the v2 contract extension
duration_ms: 27.331005
...
Subtest: gold loader rejects unsupported dataset versions
ok 131 - gold loader rejects unsupported dataset versions
duration_ms: 0.708722
...
Subtest: gold evaluator rejects unknown linking cohorts
ok 132 - gold evaluator rejects unknown linking cohorts
duration_ms: 6.01285
...
Subtest: gold v2 must declare diagnostics coverage
ok 133 - gold v2 must declare diagnostics coverage
duration_ms: 1.503149
...
Subtest: published gold schema matches the runtime contract
ok 134 - published gold schema matches the runtime contract
duration_ms: 1.821159
...
Subtest: gold evaluator rejects fixture files outside its temporary workspace
ok 135 - gold evaluator rejects fixture files outside its temporary workspace
duration_ms: 8.414502
...
Subtest: Linker connects plan, Git claim and AST fact
ok 136 - Linker connects plan, Git claim and AST fact
duration_ms: 16.304066
...
Subtest: Linker connects prose intent to a module through three grounded capability topics
ok 137 - Linker connects prose intent to a module through three grounded capability topics
duration_ms: 1.123335
...
Subtest: Linker does not connect a module on one generic topic alone
ok 138 - Linker does not connect a module on one generic topic alone
duration_ms: 1.010555
...
Subtest: An existing target path does not prove an unrelated capability
ok 139 - An existing target path does not prove an unrelated capability
duration_ms: 1.744485
...
Subtest: An existing target path plus an AST capability proves implementation
ok 140 - An existing target path plus an AST capability proves implementation
duration_ms: 1.658577
...
Subtest: Diagnostics distinguish descriptive documentation from prescriptive requirements
ok 141 - Diagnostics distinguish descriptive documentation from prescriptive requirements
duration_ms: 1.801252
...
Subtest: A changelog entry naming an extracted documentation file has release evidence
ok 142 - A changelog entry naming an extracted documentation file has release evidence
duration_ms: 1.360635
...
Subtest: Diagnostics ignore non-actionable changelog mechanics but retain release claims
ok 143 - Diagnostics ignore non-actionable changelog mechanics but retain release claims
duration_ms: 6.242388
...
Subtest: Grounded conclusion and TODO proposal contracts accept traceable values
ok 144 - Grounded conclusion and TODO proposal contracts accept traceable values
duration_ms: 7.883767
...
Subtest: Stable IDs ignore ordering noise but change with semantic content
ok 145 - Stable IDs ignore ordering noise but change with semantic content
duration_ms: 1.389677
...
Subtest: Validators reject ungrounded citations and stale semantic IDs
ok 146 - Validators reject ungrounded citations and stale semantic IDs
duration_ms: 2.044739
...
Subtest: Generation metadata exposes LLM failures instead of silently masking them
ok 147 - Generation metadata exposes LLM failures instead of silently masking them
duration_ms: 1.212347
...
Subtest: TODO proposal collections enforce dependency integrity
ok 148 - TODO proposal collections enforce dependency integrity
duration_ms: 1.255271
...
Subtest: Published JSON schemas identify all grounded output contract versions
ok 149 - Published JSON schemas identify all grounded output contract versions
duration_ms: 7.596157
...
Subtest: Blank lines and comments produce no rules
ok 150 - Blank lines and comments produce no rules
duration_ms: 1.716916
...
Subtest: A pattern without a slash matches at any depth
ok 151 - A pattern without a slash matches at any depth
duration_ms: 0.471175
...
Subtest: A leading slash anchors the pattern to the root
ok 152 - A leading slash anchors the pattern to the root
duration_ms: 0.246746
...
Subtest: A trailing slash restricts the rule to directories
ok 153 - A trailing slash restricts the rule to directories
duration_ms: 0.212838
...
Subtest: Wildcards respect path separators
ok 154 - Wildcards respect path separators
duration_ms: 0.525541
...
Subtest: Every dot-directory is excluded by
.*/ok 155 - Every dot-directory is excluded by
.*/duration_ms: 0.347759
...
Subtest: Negation re-includes a previously excluded path
ok 156 - Negation re-includes a previously excluded path
duration_ms: 0.423495
...
Subtest: Negation cannot resurrect a file inside an excluded directory
ok 157 - Negation cannot resurrect a file inside an excluded directory
duration_ms: 0.249289
...
Subtest: Last matching rule wins
ok 158 - Last matching rule wins
duration_ms: 0.352321
...
Subtest: Character classes are supported
ok 159 - Character classes are supported
duration_ms: 0.458094
...
Subtest: Paths are normalised before matching
ok 160 - Paths are normalised before matching
duration_ms: 0.338807
...
Subtest: loadIgnoreMatcher merges the three ignore files and skips missing ones
ok 161 - loadIgnoreMatcher merges the three ignore files and skips missing ones
duration_ms: 15.261195
...
Subtest: A repository without ignore files excludes nothing
ok 162 - A repository without ignore files excludes nothing
duration_ms: 1.053256
...
Subtest: The shipped .intentignore excludes build output but keeps sources
ok 163 - The shipped .intentignore excludes build output but keeps sources
duration_ms: 2.963362
...
Subtest: resolveGlobs permits one explicit .intent report without recursively scanning generated runs
ok 164 - resolveGlobs permits one explicit .intent report without recursively scanning generated runs
duration_ms: 9.655215
...
Subtest: Two unrelated AST facts sharing only a file are not linked
ok 165 - Two unrelated AST facts sharing only a file are not linked
duration_ms: 14.620301
...
Subtest: AST facts sharing a symbol are still linked despite the path rule
ok 166 - AST facts sharing a symbol are still linked despite the path rule
duration_ms: 2.20508
...
Subtest: AST details sharing only a file and generic tokens do not create a quadratic subgraph
ok 167 - AST details sharing only a file and generic tokens do not create a quadratic subgraph
duration_ms: 3.129436
...
Subtest: A file-level plan links once to the AST module aggregate instead of every detail
ok 168 - A file-level plan links once to the AST module aggregate instead of every detail
duration_ms: 7.648519
...
Subtest: A shared path still links a plan to an AST fact
ok 169 - A shared path still links a plan to an AST fact
duration_ms: 0.966149
...
Subtest: A bare filename links to a module only when its repository path is unique
ok 170 - A bare filename links to a module only when its repository path is unique
duration_ms: 1.708608
...
Subtest: A bare filename refuses ambiguous module paths
ok 171 - A bare filename refuses ambiguous module paths
duration_ms: 1.068015
...
Subtest: Relations that carry a conclusion survive alongside suppressed noise
ok 172 - Relations that carry a conclusion survive alongside suppressed noise
duration_ms: 2.265753
...
Subtest: Pair ordering stays deterministic across rebuilds
ok 173 - Pair ordering stays deterministic across rebuilds
duration_ms: 2.183748
...
Subtest: Two configuration declarations sharing only a key name are not linked
ok 174 - Two configuration declarations sharing only a key name are not linked
duration_ms: 0.533832
...
Subtest: A shared ticket still connects two configuration declarations
ok 175 - A shared ticket still connects two configuration declarations
duration_ms: 0.454136
...
Subtest: Configuration still links to documentation that describes it
ok 176 - Configuration still links to documentation that describes it
duration_ms: 0.546919
...
Subtest: Configuration file aggregate is the file-level target for an explicit documentation path
ok 177 - Configuration file aggregate is the file-level target for an explicit documentation path
duration_ms: 0.807084
...
Subtest: Configuration aggregates do not create broad capability-topic links
ok 178 - Configuration aggregates do not create broad capability-topic links
duration_ms: 0.511549
...
Subtest: a full six-stage live run passes and reports every stage
ok 179 - a full six-stage live run passes and reports every stage
duration_ms: 3.426661
...
Subtest: a stage that silently fell back to deterministic fails the check
ok 180 - a stage that silently fell back to deterministic fails the check
duration_ms: 0.517167
...
Subtest: a missing stage cannot pass as covered
ok 181 - a missing stage cannot pass as covered
duration_ms: 0.365281
...
Subtest: per-stage and total budgets are enforced separately
ok 182 - per-stage and total budgets are enforced separately
duration_ms: 0.405856
...
Subtest: live request timeout reaches the stage budget without shortening a larger override
ok 183 - live request timeout reaches the stage budget without shortening a larger override
duration_ms: 0.195128
...
Subtest: a stage reason is recorded with provider text redacted
ok 184 - a stage reason is recorded with provider text redacted
duration_ms: 0.80547
...
Subtest: history records the trend without gating on it
ok 185 - history records the trend without gating on it
duration_ms: 0.458017
...
Subtest: recorded audit history includes the current run exactly once
ok 186 - recorded audit history includes the current run exactly once
duration_ms: 0.772135
...
Subtest: history stays chronological, bounded and free of duplicate runs
ok 187 - history stays chronological, bounded and free of duplicate runs
duration_ms: 7.969106
...
Subtest: an audit converts to exactly the redacted fields history keeps
ok 188 - an audit converts to exactly the redacted fields history keeps
duration_ms: 1.174095
...
Subtest: an empty history summarizes without pretending to have measured anything
ok 189 - an empty history summarizes without pretending to have measured anything
duration_ms: 0.201234
...
Subtest: a batched run is measured per record, not per request
ok 190 - a batched run is measured per record, not per request
duration_ms: 6.317728
...
Subtest: a model whose response the validator rejected is not counted as enriched
ok 191 - a model whose response the validator rejected is not counted as enriched
duration_ms: 0.351968
...
Subtest: a failed model is a comparison result rather than a crash
ok 192 - a failed model is a comparison result rather than a crash
duration_ms: 0.89634
...
Subtest: agreement compares only records both models enriched
ok 193 - agreement compares only records both models enriched
duration_ms: 0.364976
...
Subtest: agreement is absent rather than perfect when nothing overlaps
ok 194 - agreement is absent rather than perfect when nothing overlaps
duration_ms: 0.53979
...
Subtest: the rendered comparison names the cheapest and fastest passing model
ok 195 - the rendered comparison names the cheapest and fastest passing model
duration_ms: 0.322858
...
Subtest: semantic LLM modes default to require-llm
ok 196 - semantic LLM modes default to require-llm
duration_ms: 1.889451
...
Subtest: Markdown extractor separates TODO plans and changelog claims
ok 197 - Markdown extractor separates TODO plans and changelog claims
duration_ms: 18.34059
...
Subtest: Markdown extractor preserves indented continuation lines and their source range
ok 198 - Markdown extractor preserves indented continuation lines and their source range
duration_ms: 5.822737
...
Subtest: TODO bare filenames inherit an existing directory from the heading scope
ok 199 - TODO bare filenames inherit an existing directory from the heading scope
duration_ms: 2.23172
...
Subtest: TODO resolves a bare filename only when its repository basename is unique
ok 200 - TODO resolves a bare filename only when its repository basename is unique
duration_ms: 3.861693
...
Subtest: TODO and CHANGELOG resolve the same bare filename to one repository path
ok 201 - TODO and CHANGELOG resolve the same bare filename to one repository path
duration_ms: 2.754075
...
Subtest: CHANGELOG keeps an ambiguous bare filename unresolved
ok 202 - CHANGELOG keeps an ambiguous bare filename unresolved
duration_ms: 2.601177
...
Subtest: Markdown path resolution drops paths and heading scopes outside the repository
ok 203 - Markdown path resolution drops paths and heading scopes outside the repository
duration_ms: 1.189923
...
Subtest: TODO and CHANGELOG receive audited LLM enrichment without changing structural facts
ok 204 - TODO and CHANGELOG receive audited LLM enrichment without changing structural facts
duration_ms: 25.511006
...
Subtest: Markdown enrichment corrects one rejected response and audits both attempts
ok 205 - Markdown enrichment corrects one rejected response and audits both attempts
duration_ms: 4.883962
...
Subtest: large Markdown enrichment uses bounded concurrency and keeps provider audits ordered
ok 206 - large Markdown enrichment uses bounded concurrency and keeps provider audits ordered
duration_ms: 52.81593
...
Subtest: TODO and CHANGELOG LLM fallback and require mode are explicit
ok 207 - TODO and CHANGELOG LLM fallback and require mode are explicit
duration_ms: 2.453443
...
Subtest: TODO and CHANGELOG reject structurally invalid LLM enrichments
ok 208 - TODO and CHANGELOG reject structurally invalid LLM enrichments
duration_ms: 2.612869
...
Subtest: a truncated batch is split and every record keeps its own response provenance
ok 209 - a truncated batch is split and every record keeps its own response provenance
duration_ms: 5.259736
...
Subtest: a malformed batch response splits instead of failing the whole stage
ok 210 - a malformed batch response splits instead of failing the whole stage
duration_ms: 5.600509
...
Subtest: MCP 2026 profile is stateless and exposes discovery plus complete results
ok 211 - MCP 2026 profile is stateless and exposes discovery plus complete results
duration_ms: 2.129088
...
Subtest: MCP 2026 rejects missing metadata and unsupported versions with protocol errors
ok 212 - MCP 2026 rejects missing metadata and unsupported versions with protocol errors
duration_ms: 0.790342
...
Subtest: MCP legacy profile negotiates 2025-11-25 and requires initialize
ok 213 - MCP legacy profile negotiates 2025-11-25 and requires initialize
duration_ms: 0.393414
...
Subtest: MCP exposes annotated intake command/query tools backed by the domain handler
ok 214 - MCP exposes annotated intake command/query tools backed by the domain handler
duration_ms: 12.531923
...
Subtest: An LLM record is marked as inference and keeps runtime-owned provenance
ok 215 - An LLM record is marked as inference and keeps runtime-owned provenance
duration_ms: 35.112278
...
Subtest: NL extraction corrects one rejected structured response and audits both attempts
ok 216 - NL extraction corrects one rejected structured response and audits both attempts
duration_ms: 4.343941
...
Subtest: Confidence must satisfy the provider schema instead of being silently clamped
ok 217 - Confidence must satisfy the provider schema instead of being silently clamped
duration_ms: 8.433762
...
Subtest: Source lines are clamped to the real file
ok 218 - Source lines are clamped to the real file
duration_ms: 2.602579
...
Subtest: A placeholder object is recorded as a missing field, not as content
ok 219 - A placeholder object is recorded as a missing field, not as content
duration_ms: 11.648266
...
Subtest: A real object is kept verbatim and reports no missing field
ok 220 - A real object is kept verbatim and reports no missing field
duration_ms: 2.16964
...
Subtest: The explicit unknown action is reported as a missing field
ok 221 - The explicit unknown action is reported as a missing field
duration_ms: 1.615956
...
Subtest: Both gaps are reported together
ok 222 - Both gaps are reported together
duration_ms: 1.433077
...
Subtest: Out-of-vocabulary enums are rejected instead of changing the provider intent
ok 223 - Out-of-vocabulary enums are rejected instead of changing the provider intent
duration_ms: 4.738106
...
Subtest: Rejected NL output keeps provider metadata in the failed audit
ok 224 - Rejected NL output keeps provider metadata in the failed audit
duration_ms: 2.264881
...
Subtest: The documented confidence hierarchy holds across LLM extractors
ok 225 - The documented confidence hierarchy holds across LLM extractors
duration_ms: 0.902748
...
Subtest: NL extractor produces deterministic non-LLM records
ok 226 - NL extractor produces deterministic non-LLM records
duration_ms: 10.154322
...
Subtest: NL public extraction boundary names a missing sourcePath before path resolution
ok 227 - NL public extraction boundary names a missing sourcePath before path resolution
duration_ms: 0.957022
...
Subtest: deterministic NL fallback skips Markdown headings and recognizes comparison intent
ok 228 - deterministic NL fallback skips Markdown headings and recognizes comparison intent
duration_ms: 4.197273
...
Subtest: path extraction rejects lowercase prose alternations without losing repository paths
ok 229 - path extraction rejects lowercase prose alternations without losing repository paths
duration_ms: 0.873315
...
Subtest: path extraction rejects dotted DSL fields but keeps known file extensions
ok 230 - path extraction rejects dotted DSL fields but keeps known file extensions
duration_ms: 0.263498
...
Subtest: detectModality ignores parenthetical labels and bare adjectives
ok 231 - detectModality ignores parenthetical labels and bare adjectives
duration_ms: 0.552519
...
Subtest: detectModality reads prohibitions and periphrastic obligation as requirements
ok 232 - detectModality reads prohibitions and periphrastic obligation as requirements
duration_ms: 0.562719
...
Subtest: detectPolarity does not treat without-complements as sentence negation
ok 233 - detectPolarity does not treat without-complements as sentence negation
duration_ms: 0.245307
...
Subtest: path extraction rejects HTTP routes, host paths and parent traversal
ok 234 - path extraction rejects HTTP routes, host paths and parent traversal
duration_ms: 0.297079
...
Subtest: symbol extraction rejects hostnames without losing qualified code symbols
ok 235 - symbol extraction rejects hostnames without losing qualified code symbols
duration_ms: 0.373394
...
Subtest: symbol extraction separates repository files and all-caps prose from code identifiers
ok 236 - symbol extraction separates repository files and all-caps prose from code identifiers
duration_ms: 0.210236
...
Subtest: topic keywords normalize paths, camelCase and documentation word forms
ok 237 - topic keywords normalize paths, camelCase and documentation word forms
duration_ms: 0.408341
...
Subtest: NL LLM extraction emits audited provenance and bounded DSL records
ok 238 - NL LLM extraction emits audited provenance and bounded DSL records
duration_ms: 34.182372
...
Subtest: NL LLM failure is explicit when deterministic fallback is used
ok 239 - NL LLM failure is explicit when deterministic fallback is used
duration_ms: 5.018287
...
Subtest: require-llm rejects instead of silently falling back
ok 240 - require-llm rejects instead of silently falling back
duration_ms: 0.679015
...
Subtest: OpenRouter client parses structured JSON without exposing key
ok 241 - OpenRouter client parses structured JSON without exposing key
duration_ms: 26.177383
...
Subtest: OpenRouter client preserves metadata when runtime rejects structured output
ok 242 - OpenRouter client preserves metadata when runtime rejects structured output
duration_ms: 1.736352
...
Subtest: OpenRouter client lists available models after an invalid model ID
ok 243 - OpenRouter client lists available models after an invalid model ID
duration_ms: 10.728128
...
Subtest: OpenRouter JSON timeout is not repeated as a schema fallback request
ok 244 - OpenRouter JSON timeout is not repeated as a schema fallback request
duration_ms: 0.850491
...
Subtest: OpenRouter request obeys a shared pipeline deadline without retrying
ok 245 - OpenRouter request obeys a shared pipeline deadline without retrying
duration_ms: 0.72275
...
Subtest: Documentation extractor converts OpenRouter structured output to bounded LLM records
ok 246 - Documentation extractor converts OpenRouter structured output to bounded LLM records
duration_ms: 21.878429
...
Subtest: Documentation extractor reports and enforces its chunk budget
ok 247 - Documentation extractor reports and enforces its chunk budget
duration_ms: 6.128927
...
Subtest: Documentation extractor corrects one rejected chunk and audits both responses
ok 248 - Documentation extractor corrects one rejected chunk and audits both responses
duration_ms: 5.857127
...
Subtest: Documentation extractor does not spend its correction retry on a timeout
ok 249 - Documentation extractor does not spend its correction retry on a timeout
duration_ms: 3.577088
...
Subtest: Documentation extractor exposes an audited configuration failure
ok 250 - Documentation extractor exposes an audited configuration failure
duration_ms: 0.390609
...
Subtest: Documentation extractor uses bounded concurrent OpenRouter requests
ok 251 - Documentation extractor uses bounded concurrent OpenRouter requests
duration_ms: 38.945505
...
Subtest: LLM summarizer receives graph data and preserves grounded record citations
ok 252 - LLM summarizer receives graph data and preserves grounded record citations
duration_ms: 7.219022
...
Subtest: LLM summarizer validates provider fields before creating semantic IDs
ok 253 - LLM summarizer validates provider fields before creating semantic IDs
duration_ms: 5.645427
...
Subtest: LLM summarizer diagnoses a provider that ignores the response envelope
ok 254 - LLM summarizer diagnoses a provider that ignores the response envelope
duration_ms: 3.062176
...
Subtest: LLM summarizer rejects diagnostic citations outside the supplied graph
ok 255 - LLM summarizer rejects diagnostic citations outside the supplied graph
duration_ms: 4.927535
...
Subtest: LLM summarizer prioritizes documentation over the AST payload budget
ok 256 - LLM summarizer prioritizes documentation over the AST payload budget
duration_ms: 132.334546
...
Subtest: deterministic summary presents AST module aggregates instead of low-level calls
ok 257 - deterministic summary presents AST module aggregates instead of low-level calls
duration_ms: 2.850398
...
Subtest: The summarizer grounds a fabricated record citation from its diagnostic
ok 258 - The summarizer grounds a fabricated record citation from its diagnostic
duration_ms: 4.055261
...
Subtest: The summarizer still fails when the retry fabricates a diagnostic again
ok 259 - The summarizer still fails when the retry fabricates a diagnostic again
duration_ms: 3.300453
...
Subtest: variable contracts and operation plans have deterministic content-bound IDs
ok 260 - variable contracts and operation plans have deterministic content-bound IDs
duration_ms: 5.614727
...
Subtest: every variable grants Founder read/write authority and immutable variables reject other writers
ok 261 - every variable grants Founder read/write authority and immutable variables reject other writers
duration_ms: 0.720905
...
Subtest: plans reject undeclared parameters, actor visibility gaps and payload secrets
ok 262 - plans reject undeclared parameters, actor visibility gaps and payload secrets
duration_ms: 1.363876
...
Subtest: safety-sensitive commands require a Founder decision, a human boundary and verification
ok 263 - safety-sensitive commands require a Founder decision, a human boundary and verification
duration_ms: 1.057251
...
Subtest: plan hash detects semantic tampering
ok 264 - plan hash detects semantic tampering
duration_ms: 1.046037
...
Subtest: compiler emits the exact governed envelope without an execution surface
ok 265 - compiler emits the exact governed envelope without an execution surface
duration_ms: 1.065453
...
Subtest: runtime draft boundaries ignore lifecycle and identity fields injected by untyped callers
ok 266 - runtime draft boundaries ignore lifecycle and identity fields injected by untyped callers
duration_ms: 0.609059
...
Subtest: compiler fails closed on extra, stale, wrong-source and wrong-type bindings
ok 267 - compiler fails closed on extra, stale, wrong-source and wrong-type bindings
duration_ms: 1.471806
...
Subtest: file boundary writes one private envelope atomically and refuses overwrite
ok 268 - file boundary writes one private envelope atomically and refuses overwrite
duration_ms: 13.57971
...
Subtest: Offline pipeline writes a complete run
ok 269 - Offline pipeline writes a complete run
duration_ms: 141.186458
...
Subtest: Pipeline persists synthesis, validation and review patch, then registers approval receipt
ok 270 - Pipeline persists synthesis, validation and review patch, then registers approval receipt
duration_ms: 30.003963
...
Subtest: Pipeline integrates multi-participant communication into graph, diagnostics, reality and run artifacts
ok 271 - Pipeline integrates multi-participant communication into graph, diagnostics, reality and run artifacts
duration_ms: 35.716437
...
Subtest: Pipeline require-llm task synthesis failure is audited and never publishes latest
ok 272 - Pipeline require-llm task synthesis failure is audited and never publishes latest
duration_ms: 12.271371
...
Subtest: Pipeline persists an audited failure when communication require-llm cannot run
ok 273 - Pipeline persists an audited failure when communication require-llm cannot run
duration_ms: 12.266632
...
Subtest: Pipeline persists communication stage failure and does not publish latest
ok 274 - Pipeline persists communication stage failure and does not publish latest
duration_ms: 20.504663
...
Subtest: Pipeline persists a failed manifest when NL require-llm aborts
ok 275 - Pipeline persists a failed manifest when NL require-llm aborts
duration_ms: 1.991047
...
Subtest: Pipeline persists a failed manifest when Markdown require-llm aborts
ok 276 - Pipeline persists a failed manifest when Markdown require-llm aborts
duration_ms: 8.840128
...
Subtest: Pipeline persists a failed manifest for an unexpected summary failure
ok 277 - Pipeline persists a failed manifest for an unexpected summary failure
duration_ms: 10.275784
...
Subtest: Proposal validation reports existing TODO duplicates and orders dependencies before priority
ok 278 - Proposal validation reports existing TODO duplicates and orders dependencies before priority
duration_ms: 17.973545
...
Subtest: Proposal validation rejects dependency cycles and whitespace-only criterion duplicates
ok 279 - Proposal validation rejects dependency cycles and whitespace-only criterion duplicates
duration_ms: 2.345688
...
Subtest: Python package executes the local TypeScript reality runtime without a server
ok 280 - Python package executes the local TypeScript reality runtime without a server
duration_ms: 2119.643435
...
Subtest: runtime cycle extractor turns probes, violations, drift and proposals into records
ok 281 - runtime cycle extractor turns probes, violations, drift and proposals into records
duration_ms: 18.1615
...
Subtest: measurements enter as fact and the cycle own suggestions as inference
ok 282 - measurements enter as fact and the cycle own suggestions as inference
duration_ms: 5.913802
...
Subtest: a violated expectation is negative and blocked, a healthy probe positive and verified
ok 283 - a violated expectation is negative and blocked, a healthy probe positive and verified
duration_ms: 2.619289
...
Subtest: a probe that could not run is unknown, not failing
ok 284 - a probe that could not run is unknown, not failing
duration_ms: 1.520496
...
Subtest: a watched path lets a runtime fact link to the code that produces it
ok 285 - a watched path lets a runtime fact link to the code that produces it
duration_ms: 17.560164
...
Subtest: a document that is not an autonom cycle is refused by schema, not silently empty
ok 286 - a document that is not an autonom cycle is refused by schema, not silently empty
duration_ms: 2.44982
...
Subtest: Runtime validator enforces the complete Intent DSL enum and object contract
ok 287 - Runtime validator enforces the complete Intent DSL enum and object contract
duration_ms: 4.338098
...
Subtest: Linker and remote action boundary reject malformed records before graph construction
ok 288 - Linker and remote action boundary reject malformed records before graph construction
duration_ms: 5.811759
...
Subtest: Graph validator rejects invalid relations and inconsistent statistics
ok 289 - Graph validator rejects invalid relations and inconsistent statistics
duration_ms: 2.519089
...
[t2c:a2a] listening on 127.0.0.1:36077
Subtest: diff UI and TypeScript/Python SDKs use the live backend runtime
ok 290 - diff UI and TypeScript/Python SDKs use the live backend runtime
duration_ms: 234.747876
...
Subtest: MCP/A2A action boundary rejects traversal and symlink escapes
ok 291 - MCP/A2A action boundary rejects traversal and symlink escapes
duration_ms: 24.973522
...
Subtest: bounded retrieval cannot create a relation until a grounded reranker accepts it
ok 292 - bounded retrieval cannot create a relation until a grounded reranker accepts it
duration_ms: 18.962655
...
Subtest: reranker fails closed on ungrounded quotes and more than one accepted module
ok 293 - reranker fails closed on ungrounded quotes and more than one accepted module
duration_ms: 4.074137
...
Subtest: OpenRouter reranking is required, structured and reusable only through an identity-bound cache
ok 294 - OpenRouter reranking is required, structured and reusable only through an identity-bound cache
duration_ms: 74.179342
...
Subtest: published semantic reranker schemas expose the versioned bounded contracts
ok 295 - published semantic reranker schemas expose the versioned bounded contracts
duration_ms: 1.713597
...
Subtest: provider response validation diagnoses the exact property without coercion
ok 296 - provider response validation diagnoses the exact property without coercion
duration_ms: 0.609914
...
Subtest: one structured contract emits the provider schema and parses the same value
ok 297 - one structured contract emits the provider schema and parses the same value
duration_ms: 2.484289
...
Subtest: structured parsing fails closed with the exact response path
ok 298 - structured parsing fails closed with the exact response path
duration_ms: 0.874779
...
Subtest: object uniqueness uses canonical JSON identity rather than property order
ok 299 - object uniqueness uses canonical JSON identity rather than property order
duration_ms: 0.348751
...
Subtest: a short NL symbol resolves to its only AST owner
ok 300 - a short NL symbol resolves to its only AST owner
duration_ms: 20.635731
...
Subtest: an ambiguous short NL symbol does not pretend that either AST owner is selected
ok 301 - an ambiguous short NL symbol does not pretend that either AST owner is selected
duration_ms: 3.748566
...
Subtest: an explicit path selects one owner of an otherwise ambiguous symbol
ok 302 - an explicit path selects one owner of an otherwise ambiguous symbol
duration_ms: 2.217069
...
Subtest: a qualified symbol selects its exact AST declaration without a path
ok 303 - a qualified symbol selects its exact AST declaration without a path
duration_ms: 1.081319
...
Subtest: a symbol and explicit path conflict reports the observed AST location
ok 304 - a symbol and explicit path conflict reports the observed AST location
duration_ms: 0.977542
...
Subtest: missingFields diagnostics prescribe a concrete edit for every known gap
ok 305 - missingFields diagnostics prescribe a concrete edit for every known gap
duration_ms: 0.724937
...
Subtest: Target normalization canonicalizes paths, symbols and cross-language separators
ok 306 - Target normalization canonicalizes paths, symbols and cross-language separators
duration_ms: 2.138032
...
Subtest: Qualified AST symbols align with short plan and documentation targets
ok 307 - Qualified AST symbols align with short plan and documentation targets
duration_ms: 17.368799
...
Subtest: Structured task synthesis materializes stable, grounded contracts with a complete audit
ok 308 - Structured task synthesis materializes stable, grounded contracts with a complete audit
duration_ms: 45.99265
...
Subtest: blank response-local proposal keys are rejected instead of invented by the runtime
ok 309 - blank response-local proposal keys are rejected instead of invented by the runtime
duration_ms: 6.79877
...
Subtest: prefer-llm exposes raw diagnostic actions without claiming semantic task generation
ok 310 - prefer-llm exposes raw diagnostic actions without claiming semantic task generation
duration_ms: 1.267921
...
Subtest: communication divergence is grounded in task synthesis without treating agent claims as facts
ok 311 - communication divergence is grounded in task synthesis without treating agent claims as facts
duration_ms: 5.209256
...
Subtest: task synthesis defaults to require-llm and fails explicitly without a provider
ok 312 - task synthesis defaults to require-llm and fails explicitly without a provider
duration_ms: 1.273814
...
Subtest: invalid structured LLM citations are rejected or visibly degraded according to mode
ok 313 - invalid structured LLM citations are rejected or visibly degraded according to mode
duration_ms: 10.452399
...
Subtest: task synthesis timeout is audited and never retried as a format fallback
ok 314 - task synthesis timeout is audited and never retried as a format fallback
duration_ms: 16.464353
...
Subtest: A fabricated record citation is grounded from its cited diagnostic without a retry
ok 315 - A fabricated record citation is grounded from its cited diagnostic without a retry
duration_ms: 4.844261
...
Subtest: A fabricated diagnostic still fails after the corrective retry
ok 316 - A fabricated diagnostic still fails after the corrective retry
duration_ms: 5.583099
...
Subtest: TensorFlow remains an explicit fallback when the isolated adapter is not installed
ok 317 - TensorFlow remains an explicit fallback when the isolated adapter is not installed
duration_ms: 7.394806
...
Subtest: TODO patch rendering is stable, dependency-first and excludes classified duplicates
ok 318 - TODO patch rendering is stable, dependency-first and excludes classified duplicates
duration_ms: 19.761122
...
Subtest: empty and duplicate-only results render an explicit no-op patch
ok 319 - empty and duplicate-only results render an explicit no-op patch
duration_ms: 3.543215
...
Subtest: apply rejects missing or wrong approval, stale TODO and a tampered patch
ok 320 - apply rejects missing or wrong approval, stale TODO and a tampered patch
duration_ms: 16.394695
...
Subtest: approved apply is atomic, receipt-backed and idempotent
ok 321 - approved apply is atomic, receipt-backed and idempotent
duration_ms: 9.500674
...
Subtest: service actions execute LLM propose -> render -> approved apply with scoped artifacts
ok 322 - service actions execute LLM propose -> render -> approved apply with scoped artifacts
duration_ms: 38.464243
...
Subtest: scanTree prunes ignored directories and records file signatures
ok 323 - scanTree prunes ignored directories and records file signatures
duration_ms: 16.387029
...
Subtest: diffSnapshots classifies additions, modifications and removals
ok 324 - diffSnapshots classifies additions, modifications and removals
duration_ms: 0.443609
...
Subtest: describeDelta truncates long change lists
ok 325 - describeDelta truncates long change lists
duration_ms: 0.200925
...
Subtest: An unchanged tree produces exactly one report and then stays quiet
ok 326 - An unchanged tree produces exactly one report and then stays quiet
duration_ms: 3.76991
...
Subtest: Reports are rate limited to one per interval no matter how often files change
ok 327 - Reports are rate limited to one per interval no matter how often files change
duration_ms: 26.688159
...
Subtest: A change is reported once the interval has elapsed
ok 328 - A change is reported once the interval has elapsed
duration_ms: 2.817886
...
Subtest: Ignored files never trigger a report
ok 329 - Ignored files never trigger a report
duration_ms: 4.678022
...
Subtest: A failing report is surfaced and does not stop the watcher
ok 330 - A failing report is surfaced and does not stop the watcher
duration_ms: 1.51215
...
Subtest: --no-initial-report waits for a real change
ok 331 - --no-initial-report waits for a real change
duration_ms: 1.251563
...
Subtest: Communication changes trigger watch and coalesce under the existing report rate limit
ok 332 - Communication changes trigger watch and coalesce under the existing report rate limit
duration_ms: 3.311932
...
Subtest: workflow verifier rejects duplicate top-level YAML keys
ok 333 - workflow verifier rejects duplicate top-level YAML keys
duration_ms: 77.272234
...
Subtest: workspace headline trend ignores AST-only topic and source churn
ok 334 - workspace headline trend ignores AST-only topic and source churn
duration_ms: 1.456164
...
Subtest: workspace comparison measures origin/main against uncommitted filesystem intent
ok 335 - workspace comparison measures origin/main against uncommitted filesystem intent
duration_ms: 326.429539
...
1..335
tests 335
suites 0
pass 334
fail 0
cancelled 0
skipped 1
todo 0
duration_ms 10600.724353: 335 tests, 334 pass, 0 fail, 1 local JDK skip
Follow-up failure-path validation
NotFoundErrorin ~705 ms, exactly one request,max_tokens=8192Timeoutin ~799 ms, exactly one requestnpm run verify: 335 tests, 334 pass, 0 fail, 1 local JDK skipe2e-core: passe2e-full: independently owned stalesdk/rust/Cargo.lock, expected exit 101 atcargo fetch --locked